Skip to content

[🍒][PLUGIN-1847] Add SqlServerErrorDetailsProvider#552

Merged
psainics merged 1 commit intodata-integrations:release/1.12from
cloudsufi:cherry-pick/53cfd3af30e32290c7a327947cbf56538055b212
Jan 30, 2025
Merged

[🍒][PLUGIN-1847] Add SqlServerErrorDetailsProvider#552
psainics merged 1 commit intodata-integrations:release/1.12from
cloudsufi:cherry-pick/53cfd3af30e32290c7a327947cbf56538055b212

Conversation

@psainics
Copy link
Contributor

🍒 [cherrypick]

Commits :

PR:


ErrorDetailsProvider - SqlServer [Source|Sink] plugin

Jira : PLUGIN-1847

Description

Implement Program Failure Exception Handling in SqlServer Source/Sink plugin to catch known errors

Code change

  • Added SqlServerErrorDetailsProvider.java
  • Modified DBUtils.java
  • Modified SqlServerSink.java
  • Modified SqlServerSource.java

Tests

  • Test Case (Violate check constrain)
Raw Logs

2025-01-29 09:20:32,606 - ERROR [Executor task launch worker for task 0.0 in stage 0.0 (TID 0):o.a.s.u.Utils@98] - Aborting task
io.cdap.cdap.api.exception.WrappedStageException: Stage 'SQL Server4' encountered : io.cdap.cdap.api.exception.ProgramFailureException: Error occurred in the phase: 'Writing' with sqlState: '23000', errorCode: '547', errorMessage: The INSERT statement conflicted with the CHECK constraint "CHK_Gender". The conflict occurred in database "master", table "dbo.users", column 'gender'.
  at io.cdap.cdap.etl.common.ErrorDetails.handleException(ErrorDetails.java:77)
  at io.cdap.cdap.etl.spark.io.StageTrackingRecordWriter.close(StageTrackingRecordWriter.java:67)
  at org.apache.spark.internal.io.HadoopMapReduceWriteConfigUtil.closeWriter(SparkHadoopWriter.scala:373)
  at org.apache.spark.internal.io.SparkHadoopWriter$.$anonfun$executeTask$1(SparkHadoopWriter.scala:145)
  at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538)
  at org.apache.spark.internal.io.SparkHadoopWriter$.executeTask(SparkHadoopWriter.scala:135)
  at org.apache.spark.internal.io.SparkHadoopWriter$.$anonfun$write$1(SparkHadoopWriter.scala:88)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  at org.apache.spark.scheduler.Task.run(Task.scala:136)
  at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
  at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:750)
Caused by: io.cdap.cdap.api.exception.ProgramFailureException: Error occurred in the phase: 'Writing' with sqlState: '23000', errorCode: '547', errorMessage: The INSERT statement conflicted with the CHECK constraint "CHK_Gender". The conflict occurred in database "master", table "dbo.users", column 'gender'.
  at io.cdap.cdap.api.exception.ProgramFailureException$Builder.build(ProgramFailureException.java:229)
  at io.cdap.cdap.api.exception.ErrorUtils.getProgramFailureException(ErrorUtils.java:198)
  at io.cdap.plugin.common.db.DBErrorDetailsProvider.getProgramFailureException(DBErrorDetailsProvider.java:196)
  at io.cdap.plugin.common.db.DBErrorDetailsProvider.getExceptionDetails(DBErrorDetailsProvider.java:170)
  at io.cdap.cdap.etl.common.ErrorDetails.handleException(ErrorDetails.java:75)
  ... 14 common frames omitted
Caused by: java.sql.BatchUpdateException: The INSERT statement conflicted with the CHECK constraint "CHK_Gender". The conflict occurred in database "master", table "dbo.users", column 'gender'.
  at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeBatch(SQLServerPreparedStatement.java:2101)
  at io.cdap.plugin.db.sink.ETLDBOutputFormat$1.close(ETLDBOutputFormat.java:99)
  at io.cdap.cdap.etl.spark.io.TrackingRecordWriter.close(TrackingRecordWriter.java:46)
  at io.cdap.cdap.etl.spark.io.StageTrackingRecordWriter.close(StageTrackingRecordWriter.java:65)
  ... 13 common frames omitted
2025-01-29 09:20:32,609 - WARN  [Executor task launch worker for task 0.0 in stage 0.0 (TID 0):i.c.p.d.s.ETLDBOutputFormat@106] - com.microsoft.sqlserver.jdbc.SQLServerException: The connection is closed.
  at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDriverError(SQLServerException.java:237)
  at com.microsoft.sqlserver.jdbc.SQLServerConnection.checkClosed(SQLServerConnection.java:1369)
  at com.microsoft.sqlserver.jdbc.SQLServerConnection.rollback(SQLServerConnection.java:3724)
  at io.cdap.plugin.db.sink.ETLDBOutputFormat$1.close(ETLDBOutputFormat.java:104)
  at io.cdap.cdap.etl.spark.io.TrackingRecordWriter.close(TrackingRecordWriter.java:46)
  at io.cdap.cdap.etl.spark.io.StageTrackingRecordWriter.close(StageTrackingRecordWriter.java:65)
  at org.apache.spark.internal.io.HadoopMapReduceWriteConfigUtil.closeWriter(SparkHadoopWriter.scala:373)
  at org.apache.spark.internal.io.SparkHadoopWriter$.$anonfun$executeTask$2(SparkHadoopWriter.scala:150)
  at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1549)
  at org.apache.spark.internal.io.SparkHadoopWriter$.executeTask(SparkHadoopWriter.scala:135)
  at org.apache.spark.internal.io.SparkHadoopWriter$.$anonfun$write$1(SparkHadoopWriter.scala:88)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  at org.apache.spark.scheduler.Task.run(Task.scala:136)
  at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
  at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:750)
  
POST v3/namespaces/{namespace-id}/apps/{app-id}/workflows/DataPipelineWorkflow/runs/{run-id}/classify
[
{
  "stageName": "SQL Server4",
  "errorCategory": "Plugin-'DB Integrity Constraint Violation'-'SQL Server4'",
  "errorReason": "The INSERT statement conflicted with the CHECK constraint \"CHK_Gender\". The conflict occurred in database \"master\", table \"dbo.users\", column 'gender'. For more details, see https://docs.microsoft.com/en-us/sql/relational-databases/errors-events/database-engine-events-and-errors",
  "errorMessage": "Error occurred in the phase: 'Writing' with sqlState: '23000', errorCode: '547', errorMessage: The INSERT statement conflicted with the CHECK constraint \"CHK_Gender\". The conflict occurred in database \"master\", table \"dbo.users\", column 'gender'.",
  "errorType": "USER",
  "dependency": "true",
  "errorCodeType": "SQLSTATE",
  "errorCode": "23000",
  "supportedDocumentationUrl": "https://docs.microsoft.com/en-us/sql/relational-databases/errors-events/database-engine-events-and-errors"
}
]

image

@psainics psainics self-assigned this Jan 29, 2025
@psainics psainics added the build label Jan 29, 2025
@psainics psainics marked this pull request as ready for review January 29, 2025 10:52
@psainics psainics force-pushed the cherry-pick/53cfd3af30e32290c7a327947cbf56538055b212 branch from 4a45d82 to 34e08e0 Compare January 30, 2025 04:28
@psainics psainics merged commit 4b701ad into data-integrations:release/1.12 Jan 30, 2025
8 of 10 checks passed
@psainics psainics deleted the cherry-pick/53cfd3af30e32290c7a327947cbf56538055b212 branch January 30, 2025 06:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants